Attempt to fix DuplicateAndClose_TcpServerHandler flaky test #103161

rokonec · 2024-06-07T13:10:23Z

Context:
We have noticed failure of DuplicateAndClose_TcpServerHandler test in Job https://dev.azure.com/dnceng-public/public/_build/results?buildId=696224&view=logs&j=cb76e204-ba8f-557d-eba6-9896b4f24865

Changes made (changes are isolated into commits for easier review/revert):
Ensure RemoteInvokeHandle is disposed - this is ensuring that Dispose of RemoteInvokeHandle will be called during test run, otherwise it would be called in Finalize and as it throws, it would shutdown whole xunit runner.

Make RunCommonHostLogic synchronous - after discussion with original author we decide reverting RunCommonHostLogic back to synchronous execution for simplicity reasons.

Document RemoteInvokeHandle DisposeAsync intent.

Replace namedpipe IPC by Socket IPC - as there is theory that namedpipe implementatation is root cause of this flakiness, I have decided to rule it our by replacing it by simple Socket based IPC. IPC is required because remote code has to start before socket is disconnect, as socket disconnection requires process ID of the other process.

Testing:
locally by unit tests

dotnet-policy-service · 2024-06-07T13:10:56Z

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

rokonec · 2024-06-07T13:19:16Z

Related failure log snapshot:

C:\h\w\BB30097A\w\B2DF096E\e>"C:\h\w\BB30097A\p\dotnet.exe" exec --runtimeconfig System.Net.Sockets.Tests.runtimeconfig.json --depsfile System.Net.Sockets.Tests.deps.json xunit.console.dll System.Net.Sockets.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing  
  Discovering: System.Net.Sockets.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Net.Sockets.Tests (found 1527 of 1819 test cases)
  Starting:    System.Net.Sockets.Tests (parallel test collections = on [2 threads], stop on fail = off)
    System.Net.Sockets.Tests.KeepAliveTest.Socket_KeepAlive_RetryCount_Failure [SKIP]
      Condition(s) not met: "IsWindowsBelow1703"
    System.Net.Sockets.Tests.UnixDomainSocketTest.Socket_CreateUnixDomainSocket_Throws_OnWindows [SKIP]
      Condition(s) not met: "IsSubWindows10"
Process terminated.     System.Net.Sockets.Tests.SocketDuplicationTests+Synchronous.DuplicateAndClose_TcpServerHandler(addressFamily: InterNetworkV6, sameProcess: False) [FAIL]
      System.Net.Sockets.SocketException : A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using a sendto call) no address was supplied.
      Stack Trace:
        /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.cs(1175,0): at System.Net.Sockets.Socket.Send(ReadOnlySpan`1 buffer, SocketFlags socketFlags)
        /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.cs(1170,0): at System.Net.Sockets.Socket.Send(ReadOnlySpan`1 buffer)
        /_/src/libraries/System.Net.Sockets/tests/FunctionalTests/SocketDuplicationTests.cs(348,0): at System.Net.Sockets.Tests.SocketDuplicationTests.PolymorphicTests`1.<>c__DisplayClass5_0.<<DuplicateAndClose_TcpServerHandler>g__RunCommonHostLogic|0>d.MoveNext()
        --- End of stack trace from previous location ---
        /_/src/libraries/System.Net.Sockets/tests/FunctionalTests/SocketDuplicationTests.cs(333,0): at System.Net.Sockets.Tests.SocketDuplicationTests.PolymorphicTests`1.DuplicateAndClose_TcpServerHandler(AddressFamily addressFamily, Boolean sameProcess)
        /_/src/libraries/System.Net.Sockets/tests/FunctionalTests/SocketDuplicationTests.cs(335,0): at System.Net.Sockets.Tests.SocketDuplicationTests.PolymorphicTests`1.DuplicateAndClose_TcpServerHandler(AddressFamily addressFamily, Boolean sameProcess)
        --- End of stack trace from previous location ---
UnhandledException
   at System.Environment.FailFast(System.Runtime.CompilerServices.StackCrawlMarkHandle, System.String, System.Runtime.CompilerServices.ObjectHandleOnStack, System.String)
   at System.Environment.FailFast(System.Threading.StackCrawlMark ByRef, System.String, System.Exception, System.String)
   at System.Environment.FailFast(System.String)
   at System.Runtime.EH.FallbackFailFast(System.Runtime.RhFailFastReason, System.Object)
   at System.Runtime.EH.UnhandledExceptionFailFastViaClasslib(System.Runtime.RhFailFastReason, System.Object, IntPtr, ExInfo ByRef)
   at System.Runtime.EH.DispatchEx(System.Runtime.StackFrameIterator ByRef, ExInfo ByRef)
   at System.Runtime.EH.RhThrowEx(System.Object, ExInfo ByRef)
   at Microsoft.DotNet.RemoteExecutor.RemoteInvokeHandle.Dispose(Boolean)
   at Microsoft.DotNet.RemoteExecutor.RemoteInvokeHandle.Finalize()
Unhandled exception. Xunit.Sdk.EqualException: Assert.Equal() Failure: Strings differ
           � (pos 0)
Expected: "test123!"
Actual:   ""
   at Xunit.Assert.Equal(ReadOnlySpan`1 expectedSpan, ReadOnlySpan`1 actualSpan, Boolean ignoreCase, Boolean ignoreLineEndingDifferences, Boolean ignoreWhiteSpaceDifferences, Boolean ignoreAllWhiteSpace) in /_/src/Microsoft.DotNet.XUnitAssert/src/SpanAsserts.cs:line 658
   at Xunit.Assert.Equal(String expected, String actual, Boolean ignoreCase, Boolean ignoreLineEndingDifferences, Boolean ignoreWhiteSpaceDifferences, Boolean ignoreAllWhiteSpace) in /_/src/Microsoft.DotNet.XUnitAssert/src/StringAsserts.cs:line 333
   at Xunit.Assert.Equal(String expected, String actual) in /_/src/Microsoft.DotNet.XUnitAssert/src/StringAsserts.cs:line 302
   at System.Net.Sockets.Tests.SocketDuplicationTests.PolymorphicTests`1.<DuplicateAndClose_TcpServerHandler>g__HandlerServerCode|5_1(String ipcPipeName) in /_/src/libraries/System.Net.Sockets/tests/FunctionalTests/SocketDuplicationTests.cs:line 369
   at System.Net.Sockets.Tests.SocketDuplicationTests.PolymorphicTests`1.<DuplicateAndClose_TcpServerHandler>g__HandlerServerCode|5_1(String ipcPipeName) in /_/src/libraries/System.Net.Sockets/tests/FunctionalTests/SocketDuplicationTests.cs:line 371
   at Microsoft.DotNet.RemoteExecutor.Program.Main(String[] args) in /_/src/Microsoft.DotNet.RemoteExecutor/src/Program.cs:line 61
--- End of stack trace from previous location ---
   at Microsoft.DotNet.RemoteExecutor.Program.Main(String[] args) in /_/src/Microsoft.DotNet.RemoteExecutor/src/Program.cs:line 97
----- end Mon 06/03/2024 21:26:18.67 ----- exit code -2146232797 ----------------------------------------------------------
----- start ===============  XUnitLogChecker Output =====================================================
C:\h\w\BB30097A\p\dotnet.exe --roll-forward Major C:\h\w\BB30097A\p\XUnitLogChecker.dll --dumps-path C:\cores
Unhandled exception. System.IO.EndOfStreamException: Unable to read beyond the end of the stream.
   at System.IO.Stream.ReadAtLeastCore(Span`1 buffer, Int32 minimumBytes, Boolean throwOnEndOfStream) in /_/src/libraries/System.Private.CoreLib/src/System/IO/Stream.cs:line 906
   at System.IO.BinaryReader.InternalRead(Span`1 buffer) in /_/src/libraries/System.Private.CoreLib/src/System/IO/BinaryReader.cs:line 470
   at System.IO.BinaryReader.ReadInt32() in /_/src/libraries/System.Private.CoreLib/src/System/IO/BinaryReader.cs:line 219
   at System.Net.Sockets.Tests.SocketDuplicationTests.PolymorphicTests`1.ReadSocketInfo(Stream stream) in /_/src/libraries/System.Net.Sockets/tests/FunctionalTests/SocketDuplicationTests.cs:line 281
   at System.Net.Sockets.Tests.SocketDuplicationTests.PolymorphicTests`1.<DuplicateAndClose_TcpServerHandler>g__HandlerServerCode|5_1(String ipcPipeName) in /_/src/libraries/System.Net.Sockets/tests/FunctionalTests/SocketDuplicationTests.cs:line 357
   at System.Net.Sockets.Tests.SocketDuplicationTests.PolymorphicTests`1.<DuplicateAndClose_TcpServerHandler>g__HandlerServerCode|5_1(String ipcPipeName) in /_/src/libraries/System.Net.Sockets/tests/FunctionalTests/SocketDuplicationTests.cs:line 371
   at Microsoft.DotNet.RemoteExecutor.Program.Main(String[] args) in /_/src/Microsoft.DotNet.RemoteExecutor/src/Program.cs:line 61
--- End of stack trace from previous location ---
   at Microsoft.DotNet.RemoteExecutor.Program.Main(String[] args) in /_/src/Microsoft.DotNet.RemoteExecutor/src/Program.cs:line 97
[XUnitLogChecker]: 21:26:19.09: Only dumps checking will be done.

src/libraries/System.Net.Sockets/tests/FunctionalTests/SocketDuplicationTests.cs

wfurt

LGTM

wfurt · 2024-06-07T17:15:32Z

/azp run runtime-libraries-coreclr outerloop

azure-pipelines · 2024-06-07T17:15:47Z

Azure Pipelines successfully started running 1 pipeline(s).

rokonec · 2024-06-11T15:44:32Z

/azp run runtime-libraries-coreclr outerloop

azure-pipelines · 2024-06-11T15:44:46Z

Azure Pipelines successfully started running 1 pipeline(s).

src/libraries/System.Net.Sockets/tests/FunctionalTests/SocketDuplicationTests.cs

rokonec · 2024-06-11T22:08:41Z

/azp run

azure-pipelines · 2024-06-11T22:08:46Z

You have several pipelines (over 10) configured to build pull requests in this repository. Specify which pipelines you would like to run by using /azp run [pipelines] command. You can specify multiple pipelines using a comma separated list.

rokonec · 2024-06-12T16:13:12Z

/ba-g After careful review I am quite sure failing tests are not related to changes of this PR

rokonec added 5 commits June 6, 2024 23:06

Ensure RemoteInvokeHandle is disposed

cb3a84c

Make RunCommonHostLogic synchronous

0d72795

Document DisposeAsync intent.

babfa61

Repalce namedpipe IPC by Socket IPC

922d4be

Revert .sln

666c5db

dotnet-issue-labeler bot added the area-System.Net.Sockets label Jun 7, 2024

rokonec requested a review from liveans June 7, 2024 13:10

dotnet-policy-service bot assigned rokonec Jun 7, 2024

rokonec requested review from ManickaP and MihaZupan June 7, 2024 13:10

wfurt reviewed Jun 7, 2024

View reviewed changes

src/libraries/System.Net.Sockets/tests/FunctionalTests/SocketDuplicationTests.cs Show resolved Hide resolved

wfurt approved these changes Jun 7, 2024

View reviewed changes

Setting NoDelay for ipc sockets

e61fd71

ManickaP reviewed Jun 11, 2024

View reviewed changes

src/libraries/System.Net.Sockets/tests/FunctionalTests/SocketDuplicationTests.cs Show resolved Hide resolved

build-analysis bot mentioned this pull request Jun 11, 2024

The Operation will be canceled. The next steps may not contain expected logs. dotnet/dnceng#3008

Open

3 tasks

rokonec closed this Jun 12, 2024

rokonec reopened this Jun 12, 2024

build-analysis bot mentioned this pull request Jun 12, 2024

[browser][ST] Wasm tests timeout #103246

Closed

rokonec merged commit 85a70c4 into dotnet:main Jun 12, 2024
81 of 91 checks passed

karelz added this to the 9.0.0 milestone Jun 24, 2024

github-actions bot locked and limited conversation to collaborators Jul 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Attempt to fix DuplicateAndClose_TcpServerHandler flaky test #103161

Attempt to fix DuplicateAndClose_TcpServerHandler flaky test #103161

rokonec commented Jun 7, 2024 •

edited

Loading

dotnet-policy-service bot commented Jun 7, 2024

rokonec commented Jun 7, 2024

wfurt left a comment

wfurt commented Jun 7, 2024

azure-pipelines bot commented Jun 7, 2024

rokonec commented Jun 11, 2024

azure-pipelines bot commented Jun 11, 2024

rokonec commented Jun 11, 2024

azure-pipelines bot commented Jun 11, 2024

rokonec commented Jun 12, 2024

Attempt to fix DuplicateAndClose_TcpServerHandler flaky test #103161

Attempt to fix DuplicateAndClose_TcpServerHandler flaky test #103161

Conversation

rokonec commented Jun 7, 2024 • edited Loading

dotnet-policy-service bot commented Jun 7, 2024

rokonec commented Jun 7, 2024

wfurt left a comment

Choose a reason for hiding this comment

wfurt commented Jun 7, 2024

azure-pipelines bot commented Jun 7, 2024

rokonec commented Jun 11, 2024

azure-pipelines bot commented Jun 11, 2024

rokonec commented Jun 11, 2024

azure-pipelines bot commented Jun 11, 2024

rokonec commented Jun 12, 2024

rokonec commented Jun 7, 2024 •

edited

Loading